{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3e93f3e3",
   "metadata": {},
   "source": [
    "# 2.7 Enhancing model capabilities through fine-tuning  \n",
    "\n",
    "## 🚄 Preface\n",
    "\n",
    "In the previous lessons, we introduced how to build a Q&A bot and attempted to enhance its capabilities by optimizing prompts, constructing RAG chatbot, and adding plugins. However, you may have noticed that you've been \"patching\" around the model—that is, these methods essentially enhance the model's performance through external tools, while the model's inherent knowledge boundaries and reasoning abilities remain fundamentally unchanged. This section will take you into the \"training ground\" of LLMs, directly improving the model’s underlying capabilities through fine-tuning techniques.\n",
    "\n",
    "When facing in-depth needs in specific domains, such as precise parsing of elementary school math problems, relying on prompt engineering and RAG chatbot often falls short. For details like operator precedence rules or unit conversion logic in word problems, the model needs to establish a structured knowledge system. This is where fine-tuning shows its unique advantages: by \"targeted feeding\" the model with math problem-solving examples generated by DeepSeek-R2, you can enable the model to learn DeepSeek-R2's knowledge in mathematics, grasp mathematical thinking paradigms, and even independently discover problem-solving patterns.\n",
    "\n",
    "## 🍁 Goals\n",
    "\n",
    "In this lesson you will :\n",
    "* Learn about the core principles and implementation logic of fine-tuning LLMs.\n",
    "* Combine training principles to master the methodology for optimizing key training parameters.\n",
    "* Independently complete the fine-tuning of a model, learn about potential issues that may arise, and practice various solutions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98334953",
   "metadata": {},
   "source": [
    "## 0. Environment preparation\n",
    "\n",
    "Since fine-tuning models require high hardware performance, it is recommended to use Platform for AI (PAI)'s Data Science Workshop to create an instance equipped with a GPU, allowing you to complete the fine-tuning tasks more efficiently.\n",
    "\n",
    "> If you do not have a local GPU environment or your GPU memory is less than 30 GB, it is not recommended to run this course locally, because the code may fail to execute.\n",
    "\n",
    "Please refer to \"[1_0_Setup_Computing_Environment](https://edu.aliyun.com/course/3130200/lesson/343310285)\" under `Step 1: Create a PAI DSW Instance` to create a new instance, with the following instructions:\n",
    "\n",
    "1. Ensure that the new instance has a **different name** from any previously created instance, such as acp_gpu  \n",
    "2. For **resource specifications**, select `ecs.gn7i-c8g1.2xlarge` (this specification includes **one A10 GPU with 30GB of memory**).\n",
    "<img src=\"https://img.alicdn.com/imgextra/i4/O1CN01L3iYeb1MRuEvXhhcD_!!6000000001432-2-tps-2984-1582.png\" width=\"800\">  \n",
    "3. For the **image**, choose `pytorch2.3.1`, for example `modelscope:1.29.0-pytorch2.3.1tensorflow2.16.1-gpu-py311-cu121-ubuntu22.04` (you need to switch the \"Image Configuration\" -> \"Chip Type\" to GPU).\n",
    "\n",
    "After the instance is successfully created and its status is `Running`, enter the following command in the `Terminal` to obtain the ACP course code:\n",
    "\n",
    "    ```bash\n",
    "    git clone https://github.com/AlibabaCloudDocs/aliyun_acp_learning.git\n",
    "    ```\n",
    "\n",
    "Reopen this chapter in the `Notebook` of the newly created GPU instance and continue learning the subsequent content.<br>\n",
    "\n",
    "Install the following dependencies:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "031f61d7",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "zsh:1: no matches found: ms-swift[llm]==2.4.2.post2\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "# The following dependencies need to be installed\n",
    "%pip install accelerate==1.0.1 rouge-score==0.1.2 nltk==3.9.1 ms-swift[llm]==2.4.2.post2 evalscope==0.5.5rc1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bfaa0e0",
   "metadata": {},
   "source": [
    "## 1. Task design\n",
    "\n",
    "How to solve mathematical problems has long been an important direction in the development of LLMs, and it just so happens that your intelligent assistant also needs to have basic computational capabilities. To facilitate fine-tuning of the model, select the small-parameter open-source model `qwen2.5-1.5b-instruct` as your base model.\n",
    "\n",
    "First, download the model and load it into memory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4f1a505",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-07-15T03:25:11.402851Z",
     "iopub.status.busy": "2025-07-15T03:25:11.402538Z",
     "iopub.status.idle": "2025-07-15T04:12:33.686848Z",
     "shell.execute_reply": "2025-07-15T04:12:33.686389Z",
     "shell.execute_reply.started": "2025-07-15T03:25:11.402832Z"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Download model parameters to the ./model directory\n",
    "!mkdir ./model\n",
    "!modelscope download --model qwen/Qwen2.5-1.5B-Instruct --local_dir './model'\n",
    "\n",
    "from swift.llm import (\n",
    "    get_model_tokenizer, get_template, ModelType,\n",
    "    get_default_template_type\n",
    ")\n",
    "import torch\n",
    "\n",
    "# You can modify the query (model input) according to your needs\n",
    "\n",
    "# Obtain model information\n",
    "model_type = ModelType.qwen2_5_1_5b_instruct\n",
    "template_type = get_default_template_type(model_type)\n",
    "# Set the local model location\n",
    "model_id_or_path = \"./model\"\n",
    "# Initialize the model and input/output formatting template\n",
    "kwargs = {}\n",
    "model, tokenizer = get_model_tokenizer(model_type, torch.float32, model_id_or_path=model_id_or_path, model_kwargs={'device_map': 'cpu'}, **kwargs)\n",
    "model.generation_config.max_new_tokens = 128\n",
    "template = get_template(template_type, tokenizer, default_system='')\n",
    "print(\"Model initialization completed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1063c961",
   "metadata": {},
   "source": [
    "See how it works on math problems. (Correct answer: 648 kg.)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d446abd",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-07-15T04:12:33.687820Z",
     "iopub.status.busy": "2025-07-15T04:12:33.687613Z",
     "iopub.status.idle": "2025-07-15T04:12:51.944379Z",
     "shell.execute_reply": "2025-07-15T04:12:51.943953Z",
     "shell.execute_reply.started": "2025-07-15T04:12:33.687803Z"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from swift.llm import inference\n",
    "from IPython.display import Latex, display\n",
    "\n",
    "math_question = \"In a triangular vegetable field with a base of 18 meters and a height of 6 meters, radishes are planted. If 12 kilograms of radishes are harvested per square meter, how many kilograms of radishes can be harvested from this field?\"\n",
    "query = math_question\n",
    "response, _ = inference(model, template, query)\n",
    "print(query)\n",
    "print(\"The correct answer is: 648 kilograms of radishes can be harvested\")\n",
    "print('-----------LLM response-------------')\n",
    "display(Latex(response))\n",
    "print('------------End of response--------------')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec0f6e2d",
   "metadata": {},
   "source": [
    "It seems that your model cannot accurately compute this simple mathematical problem. The model knows the formula for the area of a triangle, but fails to apply this knowledge to accurately calculate the weight of the radish.\n",
    "\n",
    "Of course, the effect of using RAG is the same. From previous learning, you know that using RAG is much like an open-book exam. However, you have never seen an open-book math exam improve scores, because the core of improving math ability lies in enhancing students' logical reasoning and computational skills, rather than knowledge retrieval.\n",
    "To enhance your Q&A bot's ability to solve simple mathematical problems, you must use model fine-tuning to improve the model’s logical reasoning ability. Computational ability can be enhanced by integrating a \"calculator\" plugin."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1dc8d595",
   "metadata": {},
   "source": [
    "## 2. Fine-tuning principles\n",
    "\n",
    "### 2.1 How models learn\n",
    "\n",
    "#### 2.1.1 Machine learning - finding patterns through data\n",
    "\n",
    "In traditional programming, you usually know the explicit rules, and write these rules into functions. For example: $f(x) = ax$.\n",
    "\n",
    "Here, $a$ is a known deterministic value (also called a parameter or weight). This function represents a simple algorithmic model that can compute (predict) the output $y$ based on the input $x$.\n",
    "\n",
    "However, in real-world scenarios, it's more likely that you don't know the explicit rules (parameters) beforehand, but you may have some observed phenomena (data).\n",
    "\n",
    "The goal of machine learning is to help you use this data (training set) to try and find (learn) these parameter values, a process known as training the model.\n",
    "\n",
    "#### 2.1.2 loss function & Cost function - quantifying model performance\n",
    "\n",
    "To find the most suitable parameters, you need a way to measure whether the currently tested parameters are appropriate.\n",
    "\n",
    "Assume you now need to evaluate whether the parameter $a$ in the model $f(x) = ax$ is suitable.\n",
    "\n",
    "##### Loss function\n",
    "\n",
    "You can assess the model's performance on a single data point $x_i, y_i$ by subtracting the predicted result $f(x_i)$ from the actual result $y_i$ for each sample $x_i$ in the training set. The function used to evaluate this error is called the loss function (or error function): $L(y_i, f(x_i)) = y_i - ax_i$.\n",
    "\n",
    "Calculating the difference might yield positive or negative values, which could cancel each other out when aggregating losses, underestimating the total loss. To address this issue, you can consider squaring the difference as the loss: $L(y_i, f(x_i)) = (y_i - ax_i)^2$. Additionally, squaring amplifies the impact of errors, helping you identify the most suitable model parameters.\n",
    "\n",
    "> In practical applications, models may use different calculation methods as the loss function.\n",
    "\n",
    "##### Cost function\n",
    "\n",
    "To evaluate the model's overall performance across the entire training set, you can calculate the average loss of all samples (that is, the mean squared error). This function, used to assess the model's overall performance across all training samples, is called the cost function.\n",
    "\n",
    "For a training set with m samples, the cost function can be expressed as: $J(a) = \\frac{1}{m} \\sum_{i=1}^{m} (y_i - ax_i)^2$.\n",
    "\n",
    "> In practical applications, different models may also choose different calculation methods as the cost function.\n",
    "\n",
    "With the cost function, the task of finding suitable model parameters can be equated to finding the minimum value of the cost function (i.e., the optimal solution). Finding the minimum value of the cost function means that the corresponding value of parameter a is the most suitable model parameter value.\n",
    "\n",
    "If you plot the cost function, the task of finding the optimal solution essentially involves finding the lowest point on the curve or surface.\n",
    "<div style=\"text-align: center\">\n",
    "<img src=\"https://img.alicdn.com/imgextra/i4/O1CN0149XTTS1WUKSTtpeoh_!!6000000002791-2-tps-2314-1682.png\" style=\"width: 400px; display: block; margin-left: auto; margin-right: auto\"/>\n",
    "</div>\n",
    "\n",
    "> In real-world projects, people often interchangeably use the terms cost function and loss function. In subsequent content and code, we will follow this engineering convention and refer to the cost function as the loss function.\n",
    "\n",
    "#### 2.1.3 Gradient descent algorithm - automatically determining the optimal solution\n",
    "\n",
    "In the previous curve, you can visually identify the lowest point. However, in practical applications, models typically have many parameters, and their loss functions are often complex surfaces in high-dimensional spaces, making it virtually impossible to find the optimal solution through direct observation. You need an automated method to find the optimal parameter configuration.\n",
    "\n",
    "Gradient descent is one of the most common methods for this. A typical implementation of gradient descent starts by randomly selecting a starting point on the surface (or curve), then continuously making small adjustments to the parameters until it finds the lowest point (corresponding to the optimal parameter configuration).\n",
    "\n",
    "<div style=\"text-align: center\">\n",
    "<img src=\"https://img.alicdn.com/imgextra/i2/O1CN01ihhR9Y1IbkFZTQ3bV_!!6000000000912-1-tps-1080-810.gif\" style=\"width: 400px;margin-left: auto; margin-right: auto\"/>\n",
    "<img src=\"https://img.alicdn.com/imgextra/i3/O1CN01meUISA1dHgq2mqm6V_!!6000000003711-1-tps-1080-810.gif\" style=\"width: 400px;margin-left: auto; margin-right: auto\"/>\n",
    "</div>\n",
    "\n",
    "When training a model, you need the training program to automatically adjust the parameters so that the value of the loss function approaches the lowest point. Therefore, the gradient descent algorithm must automatically control two aspects: the direction and the magnitude of parameter adjustment.\n",
    "\n",
    "##### Direction of parameter adjustment\n",
    "\n",
    "If the loss function is a U-shaped curve, you can intuitively see that the parameter adjustment should move in the direction where the absolute value of the slope decreases, that is, towards a flatter area.\n",
    "<div style=\"text-align: center\">\n",
    "<img src=\"https://img.alicdn.com/imgextra/i2/O1CN01ME3u6G203FVsQsmLe_!!6000000006793-2-tps-1608-1244.png\" style=\"width: 400px; display: block; margin-left: auto; margin-right: auto\"/>\n",
    "</div>\n",
    "\n",
    "If the loss function is a surface in a three-dimensional coordinate system, the direction of parameter adjustment should similarly move towards a flatter area. However, at a certain point on the surface, there are multiple possible descending directions. To find the lowest point as quickly as possible, you should move in the steepest direction.\n",
    "<div style=\"text-align: center\">\n",
    "<img src=\"https://img.alicdn.com/imgextra/i2/O1CN01Uh8OxI1mqnkBHqMjH_!!6000000005006-1-tps-664-684.gif\" style=\"width: 400px; display: block; margin-left: auto; margin-right: auto\"/>\n",
    "</div>\n",
    "\n",
    "In mathematics, the gradient points in the direction of the steepest ascent from a point on the surface, and its opposite direction is the steepest descent.\n",
    "\n",
    "To find the lowest point on the surface in the shortest time, the direction of parameter adjustment should be along the opposite direction of the gradient, denoted by the green arrow direction in the two figures above.\n",
    "\n",
    "> For a curve f(a) in a two-dimensional coordinate system, the gradient at a point is the slope at that point. \n",
    "> For a surface f(a,b) in a three-dimensional coordinate system, the gradient at a point is a two-dimensional vector composed of the slope values in the a and b axis directions. This indicates the rate of change of the function in each input variable direction and points in the direction of the fastest growth. Calculating the slope of a point on the surface in a particular axis direction is also referred to as taking the partial derivative.\n",
    "\n",
    "##### Magnitude of parameter adjustment\n",
    "\n",
    "After determining the direction of parameter adjustment, we need to determine the magnitude of the adjustment.\n",
    "\n",
    "Adjusting parameters with a fixed step size is the easiest approach, but this may prevent you from ever finding the lowest point, instead causing oscillation near the lowest point.\n",
    "\n",
    "In the figure below, adjusting parameters with a fixed step size of 1.5 results in oscillation around the lowest value, but it's unable to reach the lowest point.\n",
    "\n",
    "<div style=\"text-align: center\">\n",
    "<img src=\"https://img.alicdn.com/imgextra/i3/O1CN01y7FatQ27bKI9CYCJ1_!!6000000007815-1-tps-938-646.gif\" style=\"width: 400px; display: block; margin-left: auto; margin-right: auto\"/>\n",
    "</div>\n",
    "\n",
    "To avoid this issue, the adjustment magnitude should be reduced as you approach the lowest point. The closer you get to it, the smaller the slope should become. Instead of using a fixed step size, you can use the slope at the current position as the adjustment magnitude.\n",
    "\n",
    "<div style=\"text-align: center\">\n",
    "<img src=\"https://img.alicdn.com/imgextra/i3/O1CN01h45Ifb1xRZhXXIXEC_!!6000000006440-1-tps-892-618.gif\" style=\"width: 400px; display: block; margin-left: auto; margin-right: auto\"/>\n",
    "</div>\n",
    "\n",
    "However, some loss function curves are very steep, and simply using the slope may still cause oscillation around the lowest point. To address this, multiply the slope by a coefficient to regulate the step size. This coefficient is called the learning rate.\n",
    "\n",
    "The choice of learning rate is particularly important for training effectiveness and efficiency:\n",
    "\n",
    "<div style=\"display: flex; justify-content: space-between; gap: 2px; padding: 15px; background:rgba(0,0,0,0)\">\n",
    "    <!-- Column 1 -->\n",
    "    <div style=\"flex: 1; padding: 10px; border: 1px solid #ddd; border-radius: 5px\">\n",
    "        <p style=\"margin-top: 10px\">An appropriate learning rate allows you to find suitable parameters in a short time.</p>\n",
    "        <img src=\"https://img.alicdn.com/imgextra/i3/O1CN01NrvVfj1sCqtKHLyia_!!6000000005731-2-tps-1680-1224.png\" style=\"width: 100%; height: auto; border-radius: 3px\"/>\n",
    "    </div>\n",
    "    <!-- Column 2 -->\n",
    "    <div style=\"flex: 1; padding: 10px; border: 1px solid #ddd; border-radius: 5px\">\n",
    "        <p style=\"margin-bottom: 10px\">An excessively low learning rate, while capable of finding suitable parameters, leads to greater time and resource consumption.</p>\n",
    "        <img src=\"https://img.alicdn.com/imgextra/i1/O1CN015dbcz61MCn8LkN2Ta_!!6000000001399-2-tps-1728-1300.png\" style=\"width: 100%; height: auto; border-radius: 3px\"/>\n",
    "    </div>\n",
    "    <!-- Column 3 -->\n",
    "    <div style=\"flex: 1; padding: 10px; border: 1px solid #ddd; border-radius: 5px\">\n",
    "        <p style=\"margin-bottom: 10px\">Meanwhile, an excessively high learning rate may cause you to skip the optimal solution, ultimately failing to find the lowest point.</p>\n",
    "        <img src=\"https://img.alicdn.com/imgextra/i1/O1CN01l4leTB1LKI0BcVs16_!!6000000001280-2-tps-1658-1262.png\" style=\"width: 100%; height: auto; border-radius: 3px\"/>\n",
    "    </div>\n",
    "</div>\n",
    "</div>\n",
    "\n",
    "A smaller learning rate, although more computationally expensive and time-consuming, enables the optimization process to converge more precisely toward the minimum of the cost function. In practical model training, dynamic adjustment of the learning rate is commonly employed to balance convergence speed and accuracy. For example,  Model Studio's model fine-tuning feature includes built-in [learning rate adjustment strategy](https://help.aliyun.com/zh/model-studio/user-guide/using-fine-tuning-on-console#7864d6a606ztg), allowing users to configure linear or curve-based decay. Alibaba Cloud's PAI also provides an [AutoML](https://help.aliyun.com/zh/pai/user-guide/automl/) tool that can automatically search for an optimal learning rate, reducing manual tuning effort and improving training efficiency.\n",
    "\n",
    "#### 2.1.4 Key Hyperparameters in Model Training\n",
    "\n",
    "##### Batch size\n",
    "\n",
    "During the optimization process, each iteration—comprising gradient computation and parameter update—is referred to as a training step.\n",
    "\n",
    "In basic gradient descent, the gradient is computed using a single data point or the entire dataset. However, in practice, we typically use mini-batch gradient descent , where the batch size n determines how many samples are used per step. The gradients from these n samples are averaged before updating the model parameters.\n",
    "\n",
    "* A larger batch size accelerates training by leveraging parallel computing capabilities and stabilizing gradient estimates, but it requires more memory and computational resources.\n",
    "* However, an excessively large batch size may lead to poorer generalization due to reduced stochasticity and sharper minima.\n",
    "\n",
    "Selecting an appropriate batch size involves balancing hardware constraints, training speed, and model performance. In practice, empirical experimentation is often necessary to identify the optimal batch size for a given task.\n",
    "\n",
    "##### Evaluation steps\n",
    "\n",
    "Given the large size of most training datasets, it is inefficient to evaluate the model on the validation set after every full pass through the data. Instead, evaluation is typically performed at regular intervals during training—specifically, after every fixed number of training steps.\n",
    "\n",
    "This interval is controlled by the `eval_steps` parameter. For instance, setting `eval_steps=500` means the model will be evaluated on the validation set every 500 training steps. This approach allows for timely monitoring of model performance without significantly slowing down training.\n",
    "\n",
    "##### Epochs\n",
    "\n",
    "One complete pass through the entire training dataset is defined as an epoch. Since a single epoch is rarely sufficient to reach the optimal solution (that is, the minimum of the loss function), most training frameworks allow configuration of the total number of epochs—for example, via the `num_train_epochs` parameter in the Swift training framework.\n",
    "\n",
    "* Too few epochs may result in underfitting, where the model fails to learn the underlying patterns.\n",
    "* Too many epochs can lead to overfitting, extended training times, and unnecessary resource consumption.\n",
    "\n",
    "A widely adopted strategy to determine the ideal stopping point is early stopping:\n",
    "\n",
    "* Rather than fixing the number of epochs in advance, training continues until model performance on the validation set plateaus or begins to degrade.\n",
    "* This method helps prevent overfitting and improves efficiency.\n",
    "\n",
    "While early stopping is effective, it is not the only approach. Other techniques include adaptive learning rate scheduling based on validation loss trends, which indirectly influences the effective number of training epochs.\n",
    "\n",
    "#### 2.1.5 Neural network - a universal complex function approximator\n",
    "\n",
    "**Challenges in machine learning**\n",
    "\n",
    "In complex tasks such as text generation, both input x and output y are typically high-dimensional, making it difficult to discern explicit patterns or relationships from raw data.\n",
    "\n",
    "To address this challenge, researchers have turned to neural networks—particularly deep (multi-layer) architectures—as powerful tools for modeling intricate, non-linear mappings.\n",
    "\n",
    "<div style=\"text-align: center\">\n",
    "<img src=\"https://img.alicdn.com/imgextra/i1/O1CN01QRD5MH1rwMdJHBzxi_!!6000000005695-2-tps-1080-533.png\" style=\"width: 400px; display: block; margin-left: auto; margin-right: auto\"/>\n",
    "</div>\n",
    "\n",
    "One layer of a neural network is generally expressed as $Y=σ(W⋅X)$, where the uppercase input $X$ and output $Y$ indicate they are multi-dimensional, $σ$ is the activation function, and $W$ represents the parameters of the assumed function $f$. A k-layer neural network can be expressed as $Y=σ(W_k ⋯ σ(W_2 ⋅σ(W_1⋅X)))$.\n",
    "\n",
    "The activation function is a key component in neural networks that introduces non-linear transformations and determines whether neurons are activated and transmit information. For example, the most commonly used activation function RELU can be written as:\n",
    "\n",
    "**$RELU(input) = max( 0, input)= \\begin{cases} input & \\text{if } input > 0 \\\\ 0 & \\text{if } input ≤ 0 \\end{cases}$**\n",
    "\n",
    "When $input≤0$, the neuron is not activated; when $input>0$, the neuron is activated and begins transmitting information to the output.\n",
    "\n",
    "Expanding one layer of a neural network can be written as follows (assuming $X$ is a $3×2$ dimensional matrix and $Y$ is a $2×2$ dimensional matrix):\n",
    "\n",
    "$σ(W_{2×3}⋅X_{3×2})= σ(\\left[ \\begin{matrix} w_{1,1} & w_{1,2} & w_{1,3} \\\\ w_{2,1} & w_{2,2} & w_{2,3} \\end{matrix} \\right]×\\left[ \\begin{matrix} x_{1,1}& x_{1,2}\\\\ x_{2,1}& x_{2,2} \\\\ x_{3,1}& x_{3,2} \\end{matrix} \\right])$\n",
    "\n",
    "$= σ(\\left[\\begin{matrix}\n",
    "w_{1,1}×x_{1,1}+w_{1,2}×x_{2,1}+w_{1,3}×x_{3,1}&\n",
    "w_{1,1}×x_{1,2}+ w_{1,2}×x_{2,2}+w_{1,3}×x_{3,2} \\\\\n",
    "w_{2,1}×x_{1,1}+ w_{2,2}×x_{2,1}+w_{2,3}×x_{3,1}&\n",
    "w_{2,1}×x_{1,2}+ w_{2,2}×x_{2,2}+w_{2,3}×x_{3,2} \\end{matrix} \\right])$\n",
    "\n",
    "$= \\left[ \\begin{matrix} max(0, \\sum\\limits_{k=1}^{3}w_{1,k}×x_{k,1})& max(0, \\sum\\limits_{k=1}^{3}w_{1,k}×x_{k,2})\\\\ max(0, \\sum\\limits_{k=1}^{3}w_{2,k}×x_{k,1})& max(0, \\sum\\limits_{k=1}^{3}w_{2,k}×x_{k,2}) \\end{matrix} \\right]= \\left[ \\begin{matrix} y_{1,1}& y_{1,2}\\\\ y_{2,1}& y_{2,2} \\end{matrix} \\right]=Y_{2×2}$\n",
    "\n",
    "Fortunately, the gradient descent method remains effective on high-dimensional, complex functions.\n",
    "\n",
    "<div style=\"text-align: center\">\n",
    "<img src=\"https://img.alicdn.com/imgextra/i3/O1CN011caxP31GiUrEv1aGH_!!6000000000656-2-tps-847-779.png\" style=\"width: 400px; display: block; margin-left: auto; margin-right: auto\"/>\n",
    "</div>\n",
    "\n",
    "Now you have the winning combination:\n",
    "\n",
    "**A powerful function approximator—the neural network, capable of modeling virtually any complex, non-linear relationship—paired with an efficient optimization algorithm—gradient descent, designed to learn optimal model parameters by minimizing the loss function.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "576f0801",
   "metadata": {},
   "source": [
    "### 2.2 Efficient fine-tuning techniques\n",
    "\n",
    "#### 2.2.1 Pre-training and fine-tuning\n",
    "\n",
    "From earlier discussions, you've learned that the core of model training lies in finding an optimal set of parameters that minimize the loss function.\n",
    "\n",
    "The model you initially download—such as `qwen2.5-1.5b-instruct`—is not randomly initialized; it consists of pre-trained parameters obtained through extensive training on large-scale, diverse datasets. This pre-training equips the model with broad linguistic understanding and general reasoning capabilities.\n",
    "\n",
    "Fine-tuning refers to the process of further adjusting these pre-trained parameters using a smaller, task-specific dataset, enabling the model to specialize in particular applications—such as solving math problems, answering domain-specific questions, or generating legal documents.\n",
    "\n",
    "To appreciate the value of this two-stage approach, let’s examine what it would take to train such a model from scratch , using `qwen2.5-1.5b-instruct` (a 1.5-billion-parameter model) as an example.\n",
    "\n",
    "---\n",
    "\n",
    "#### GPU memory requirements\n",
    "\n",
    "* Memory occupied by 1.5 billion parameters (assuming FP32 precision, with each parameter taking 4 bytes):  \n",
    "    $ \\frac{1.5 \\times 10^9 \\times 4}{2^{30}} \\approx 5.59 \\text{ GB} $\n",
    "\n",
    "* In practice, training a model typically requires **7–8 times** the memory of its parameter size due to gradients, optimizer states, and intermediate activations. This brings the total GPU memory requirement to around **45 GB**, which exceeds most consumer-grade GPUs, and even many cloud-based experimental environments.\n",
    "\n",
    "---\n",
    "\n",
    "#### Training time estimation\n",
    "\n",
    "*   Example calculation:  \n",
    "    - Total training tokens = **200 billion**, or about 250,000 copies of Shakespeare's complete works\n",
    "    - Batch size (using 8 GPUs in parallel) = **2,000 tokens per batch**  \n",
    "    - Throughput = **150 tokens/GPU/sec × 8 GPUs = 1,200 tokens/sec**\n",
    "\n",
    "*   Estimated training time =  \n",
    "    $$\n",
    "    \\frac{\\text{Total Tokens}}{\\text{Batch Size} \\times \\text{Tokens per Second} \\times 86400} \\approx 10 \\text{ days}\n",
    "    $$\n",
    "\n",
    "*   Real-world considerations:  \n",
    "    Include data preprocessing, checkpoint saving, and communication overhead in distributed training can increase actual training time by **20–50%**. With larger datasets—such as 1 trillion tokens—training duration could extend to over a year, even with optimization.\n",
    "\n",
    "---\n",
    "\n",
    "#### Training cost overview\n",
    "\n",
    "*   For short-term training (such as 10-day training runs), renting cloud GPU instances on a pay-as-you-go basis is often more cost-effective than purchasing dedicated hardware .\n",
    "*   Training cost formula:  \n",
    "    $$\n",
    "    \\text{Training Cost} = \\text{GPU hourly rate} \\times \\text{Training time (in hours)}\n",
    "    $$\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "In summary, **reducing server unit price** and **shortening training time** can effectively reduce training costs, where **reducing memory requirements** lower server unit price, and reducing **total training data volume** shorten training time.\n",
    "\n",
    "<br/>\n",
    "\n",
    "One practical challenge in the actual model training process is **the high cost of obtaining labeled data, especially for specific tasks** (such as medical image analysis or niche language processing). You can try step-by-step training of the model through \"pre-training\" and \"fine-tuning\":\n",
    "\n",
    "* **Pre-training**: Training the model on a large-scale **general dataset** so that it can learn broad foundational knowledge or feature representations. This knowledge is usually general and not aimed at any specific task. Pre-training is not task-specific, but provides a powerful initial model for various downstream tasks. Typical pre-trained models: Qwen2.5-Max, DeepSeek-V3, GPT-4.\n",
    "* **Fine-tuning**: Further training the model using a **small-scale dataset** specific to a task based on the pre-trained model. The goal is to make the model adapt to specific downstream tasks (such as medical, legal, and other professional domain needs).\n",
    "\n",
    "The table below shows the main differences between pre-training and fine-tuning:\n",
    "\n",
    "<div style=\"width: 20%\">\n",
    "    \n",
    "|  **Feature**  |  **Pre-training**  |  **Fine-tuning**  |\n",
    "| --- | --- | --- |\n",
    "|  Objective  |  $ $ Learning general features  |  Adapting to specific tasks  |\n",
    "|  Data  |  Large-scale general data  |  Small-scale task-related data  |\n",
    "|  Training method  |  Self-supervised/Unsupervised  |  Supervised  |\n",
    "|  Parameter updates  |  All parameters trainable  |  Partial or all parameters trainable  |\n",
    "|  Application scenarios  |  Base model construction  |  Specific task optimization  |\n",
    "\n",
    "</div>\n",
    "\n",
    "It is worth mentioning that **pre-training generally learns through self-supervised learning**, with data coming from massive texts available on the internet (including Wikipedia, books, and web pages), allowing the model to find patterns or \"guess \" on its own. This learning method does not require manual annotation, saving significant labor costs and making it naturally suitable for learning from massive data.\n",
    "\n",
    "In contrast, **fine-tuning is done through supervised learning**, requiring small-scale annotated data for specific tasks (such as annotated reviews for sentiment classification, or annotated medical texts), and directly teaching the model to complete tasks using annotated data. Due to the high cost of manual annotation, this learning method is difficult to scale to massive data, thus making it more suitable for model training with clear scenario goals, typically requiring only a few thousand to tens of thousands of samples.\n",
    "\n",
    "Therefore, you can quickly and cost-effectively build your LLM application as follows:\n",
    "\n",
    "Step 1: Choose a pre-trained model (Qwen, DeepSeek, GPT). This can save the comprehensive cost of training a model from scratch.\n",
    "\n",
    "Step 2: Fine-tune the model according to your actual scenario, usually only needing to build a few thousand annotated data applicable to the actual scenario, because the total number of training tokens is greatly reduced. This effectively shortens the training time, thereby further reducing the training cost.\n",
    "\n",
    "Fine-tuning can shorten training time, but can fine-tuning the model also reduce memory requirements?\n",
    "\n",
    "The number of model parameters is the main factor affecting memory requirements. From the perspective of adjusting the size of the parameter count, fine-tuning can be divided into **full-parameter** and **parameter-efficient fine-tuning**.\n",
    "\n",
    "**Full-parameter fine-tuning**, also known as **full fine-tuning**, is an optimization approach that updates all parameters of a pre-trained model during adaptation to a downstream task. In this method, every layer and weight in the model architecture is trainable and can be adjusted based on task-specific data.\n",
    "\n",
    "This strategy offers two key advantages:\n",
    "\n",
    "* It avoids the prohibitive cost of training a model from scratch by leveraging powerful pre-trained representations.\n",
    "* It ensures no part of the model is frozen, reducing the risk of performance degradation due to under-adapted components.\n",
    "\n",
    "However, despite its effectiveness, full fine-tuning remains computationally expensive, especially for large-scale models. It demands:\n",
    "\n",
    "* Significant GPU memory (for storing gradients and optimizer states),\n",
    "* High-throughput hardware,\n",
    "* Long training times,\n",
    "* Large volumes of high-quality labeled data.\n",
    "\n",
    "As a result, while full fine-tuning often achieves strong performance, it is not always practical—particularly in resource-constrained environments.\n",
    "To address these challenges, **Parameter-Efficient Fine-Tuning (PEFT)** methods have emerged as a powerful alternative. PEFT techniques adapt large pre-trained models by updating only a small subset of parameters, while keeping the majority of the original model weights frozen. Remarkably, they achieve performance close to full fine-tuning—with drastically reduced computational and storage costs.\n",
    "\n",
    "Popular PEFT methods include:\n",
    "\n",
    "* Adapter Tuning: Inserts small neural network modules (\"adapters\") between transformer layers.\n",
    "* Prompt Tuning: Learns task-specific soft prompts (continuous embeddings) while freezing the model.\n",
    "* LoRA (Low-Rank Adaptation): A leading method that approximates weight changes using low-rank matrices—requiring only 0.1% to 1% of the original model’s parameters to be trained.\n",
    "\n",
    "Among these, LoRA has become the preferred choice in many real-world applications, particularly when compute, memory, or budget are limited."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f59f270",
   "metadata": {},
   "source": [
    "#### 2.2.2 LoRA fine-tuning\n",
    "\n",
    "LoRA fine-tuning is currently the most commonly used method for model adaptation. It does not rely on the internal architecture of the model but instead abstracts and decomposes the parameters that need updating during fine-tuning into two much smaller low-rank matrices $A_{d \\times r}$ and $B_{r \\times d}$. The original model weights remain frozen:\n",
    "$$\n",
    "W^{fine-tuned}_{d \\times d} = A_{d \\times r} \\cdot B_{r \\times d} + W^{pre-trained}_{d \\times d}\n",
    "$$\n",
    "\n",
    "To clarify the concept of low-rank decomposition, let's revisit a simple neural network formulation. Assume the input vector $X$ has dimension 4 and the output vector $Y$ has dimension 5. Then, the weight matrix $W$ would be of size $5 \\times 4$, denoted as $W \\in \\mathbb{R}^{5 \\times 4}$, containing a total of 20 parameters.\n",
    "\n",
    "A single-layer neural network can be expressed as:  \n",
    "$$\n",
    "Y_{5 \\times 1} = \\sigma(W_{5 \\times 4} \\cdot X_{4 \\times 1})\n",
    "$$\n",
    "\n",
    "The rank of a matrix intuitively represents its effective information content. For example, although the following matrix has 2 rows and 3 columns, all rows are linearly dependent — one row can represent the others — so its rank is 1:\n",
    "$$\n",
    "\\text{rank}\\left( \n",
    "\\begin{bmatrix}\n",
    "1 & 2 & 3 \\\\\n",
    "2 & 4 & 6\n",
    "\\end{bmatrix}\n",
    "\\right) = 1\n",
    "$$\n",
    "\n",
    "In model fine-tuning, it can be assumed that most of the useful information updates (high-rank) have already been learned during pre-training, while the additional effective information introduced by fine-tuning is minimal (low-rank). This can be written mathematically as:\n",
    "\n",
    "$$\n",
    "W_{5 \\times 4}^{pre-trained} - W_{5 \\times 4}^{initial} = \\Delta W_{5 \\times 4}^{pre-trained}, \\quad \\text{rank}(\\Delta W_{5 \\times 4}^{pre-trained}) = 5\n",
    "$$\n",
    "$$\n",
    "W_{5 \\times 4}^{fine-tuned} - W_{5 \\times 4}^{pre-trained} = \\Delta W_{5 \\times 4}^{fine-tuning}, \\quad \\text{rank}(\\Delta W_{5 \\times 4}^{fine-tuning}) \\leq 2\n",
    "$$\n",
    "\n",
    "Since low-rank matrices contain sparse information, they can be efficiently decomposed into two much smaller matrices. Assuming $\\text{rank}(\\Delta W_{5 \\times 4}^{fine-tuning}) = 1$, we can write:\n",
    "\n",
    "$$\n",
    "\\Delta W_{5 \\times 4}^{fine-tuning} =\n",
    "\\begin{bmatrix}\n",
    "1 & 0 & 2 & -1 \\\\\n",
    "2 & 0 & 4 & -2 \\\\\n",
    "3 & 0 & 6 & -3 \\\\\n",
    "4 & 0 & 8 & -4 \\\\\n",
    "5 & 0 & 10 & -5\n",
    "\\end{bmatrix}_{5 \\times 4}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    "1 \\\\\n",
    "2 \\\\\n",
    "3 \\\\\n",
    "4 \\\\\n",
    "5\n",
    "\\end{bmatrix}_{5 \\times 1}\n",
    "\\times\n",
    "\\begin{bmatrix}\n",
    "1 & 0 & 2 & -1\n",
    "\\end{bmatrix}_{1 \\times 4}\n",
    "$$\n",
    "\n",
    "To further illustrate this, consider the base model `qwen2.5-1.5b-instruct`, where we assume $r = 8$ and $d = 1024$. Below is a comparison of parameter counts:\n",
    "\n",
    "$$\n",
    "W^{fine-tuned}_{d \\times d} = A_{d \\times r} \\cdot B_{r \\times d} + W^{pre-trained}_{d \\times d}\n",
    "$$\n",
    "\n",
    "| **Method** | **Parameter Calculation Formula** | **Number of Parameters** | **Savings Ratio** |\n",
    "| --- | --- | --- | --- |\n",
    "| Full-parameter fine-tuning | $W_{d \\times d}$, $1024 \\times 1024$ | 1,048,576 | $0\\%$ |\n",
    "| LoRA fine-tuning | $A_{d \\times r}$ and $B_{r \\times d}$, $1024 \\times 8 + 8 \\times 1024$ | 16,384 | $98.44\\%$ |\n",
    "\n",
    "During inference, the matrices $A_{d \\times r}$, $B_{r \\times d}$, and $W^{pre-trained}_{d \\times d}$ can be merged to reconstruct $W^{fine-tuned}_{d \\times d}$ either in advance or dynamically.\n",
    "\n",
    "<div style=\"text-align: center;\">\n",
    "<a href=\"https://img.alicdn.com/imgextra/i3/O1CN01NtGavS1TTvIIeZxO1_!!6000000002384-2-tps-804-712.png\" target=\"_blank\">\n",
    "<img src=\"https://img.alicdn.com/imgextra/i3/O1CN01NtGavS1TTvIIeZxO1_!!6000000002384-2-tps-804-712.png\" style=\"width: 600px;background:white;display: block; margin-left: auto; margin-right: auto\"/>\n",
    "</a>\n",
    "<br>Image source: LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS\n",
    "</div>\n",
    "\n",
    "When using LoRA for fine-tuning, the main tunable hyperparameter is the assumed low-rank $r$. A larger $r$ allows the model to capture more complex feature changes but increases training difficulty, requiring more memory and training epochs.\n",
    "\n",
    "Empirically, the value of $r$ is closely related to the amount of training data:\n",
    "\n",
    "- **For small datasets (1k–10k samples):** It is recommended to use $r \\leq 16$ to prevent overfitting and excessive training time.\n",
    "- **For large datasets (100k+ samples):** Try $r \\geq 32$ to better explore underlying patterns in the data.\n",
    "\n",
    "#### 2.2.3 Effectiveness of LoRA Fine-tuning\n",
    "\n",
    "The creators of LoRA compared various fine-tuning methods across two datasets. As shown below, LoRA provides the best cost-effectiveness trade-off (the x-axis shows the number of trainable parameters, and the y-axis indicates training effectiveness).\n",
    "\n",
    "<div style=\"text-align: center;\">\n",
    "<img src=\"https://img.alicdn.com/imgextra/i1/O1CN01RGquUv1ZlDuoik8zU_!!6000000003234-2-tps-1944-662.png\" style=\"width: 700px;background:white;display: block; margin-left: auto; margin-right: auto\"/>\n",
    "</div>\n",
    "\n",
    "It is clear that not all methods benefit from having more trainable parameters — **more parameters do not necessarily lead to better performance.** However, **the LoRA method demonstrates superior scalability and task performance.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1204e108",
   "metadata": {},
   "source": [
    "## 3. Fine-tuning practice\n",
    "\n",
    "### 3.1 Model training status and metrics\n",
    "\n",
    "Training a model is very similar to the human learning and exam process.\n",
    "\n",
    "A model must be evaluated on three sets of data and generates two key metrics to determine its training status:\n",
    "\n",
    "Three sets of questions:\n",
    "\n",
    "*   **Training set**: Like a practice workbook with detailed answer explanations. The model  repeatedly learns from this data and computes the training loss based on the loss function. A smaller **training loss** means the model performs better on the training set. Combined with the gradient descent method discussed in Section 2.1, the model updates its parameters based on this loss.\n",
    "    \n",
    "*   **Validation set**: This set is like simulated exam questions. After a period of learning, the model is tested on this data to compute the validation loss, which evaluates how well the training is progressing. A lower validation loss indicates better performance on unseen but similar data.\n",
    "    \n",
    "*   **Test set**: Represents real exam questions. The model's accuracy  on the test set is used to evaluate its final, overall performance.\n",
    "\n",
    "The three states of model training:\n",
    "\n",
    "*   **Training loss stays unchanged or increases**: This indicates **training failure**. It’s as if the model isn’t learning from the practice workbook, suggesting a problem with the learning setup—such as an inappropriate learning rate or optimization issue.\n",
    "    \n",
    "*   **Both training loss and validation loss are decreasing**: This means the model is **underfitting**. The model is making progress on both the training and validation sets, but it hasn't yet fully captured the underlying patterns. In this case, you should allow training to continue.\n",
    "    \n",
    "*   **Training loss decreases while validation loss increases**: This signals overfitting. The model is essentially memorizing the training data (like memorizing answers in a workbook) rather than learning generalizable patterns. When faced with new questions (validation set), it performs poorly. To address this, techniques that discourage memorization should be applied—such as increasing data diversity (for example, by using more varied workbooks) so the model must learn the core concepts instead of rote recall."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23363db8",
   "metadata": {},
   "source": [
    "### 3.2 Baseline model examination\n",
    "\n",
    "Before beginning model fine-tuning, let's first examine how the baseline model performs on the test set.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a54a02d5",
   "metadata": {
    "ExecutionIndicator": {
     "show": true
    },
    "execution": {
     "iopub.execute_input": "2025-07-15T04:55:06.106090Z",
     "iopub.status.busy": "2025-07-15T04:55:06.105776Z",
     "iopub.status.idle": "2025-07-15T04:58:09.714944Z",
     "shell.execute_reply": "2025-07-15T04:58:09.714489Z",
     "shell.execute_reply.started": "2025-07-15T04:55:06.106071Z"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "import json\n",
    "from IPython.display import Markdown\n",
    "\n",
    "sum, score = 0, 0\n",
    "for line in open(\"./resources/2_7/test.jsonl\"):\n",
    "    # Read math questions from the test set\n",
    "    math_question = json.loads(line)\n",
    "    query = math_question[\"messages\"][1][\"content\"]\n",
    "    # Inference using the baseline model\n",
    "    response, _ = inference(model, template, query)\n",
    "    # Get the correct answer\n",
    "    ans = math_question[\"messages\"][2][\"content\"]\n",
    "    pos = ans.find(\"ans\")\n",
    "    end_pos = ans[pos:].find('}}')\n",
    "    ans = ans[pos - 2: end_pos + pos + 2]\n",
    "    # Format output\n",
    "    print((\"========================================================================================\"))\n",
    "    print(query.split(\"#Math Problem#\\n\")[1])\n",
    "    print(\"The correct answer is: \" + ans)\n",
    "    print(\"-----------Model Response----------------\")\n",
    "    display(Latex(response))\n",
    "    print(\"-----------End of Response----------------\")\n",
    "    # Calculate model score\n",
    "    if ans in response or ans[6:-2] in response:\n",
    "        score += 1\n",
    "        print(\"Model answered correctly\")\n",
    "    else: print(\"Model answered incorrectly\")\n",
    "    sum += 1\n",
    "# Summary\n",
    "display(Markdown(\"Model scored: **\" + str(int(100*score/sum)) + \"** points in the exam\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4baa331f",
   "metadata": {},
   "source": [
    "The baseline model often gives up reasoning midway during exams and struggles to produce correct answers. This behavior not only confirms that the questions exceed its capability but also reveals why prompt engineering alone fails: the model lacks the fundamental problem-solving skills needed to tackle such tasks. In this case, model fine-tuning is the only effective solution."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bbf59e5e",
   "metadata": {},
   "source": [
    "### 3.3 Model fine-tuning\n",
    "\n",
    "In this section, we will use the [ms-swift](https://github.com/modelscope/ms-swift/tree/main) (Modelscope Scalable lightWeight Infrastructure for Fine-Tuning) framework—an open-source framework specifically developed by Alibaba's ModelScope community for efficient model training. \n",
    "\n",
    "This framework supports a full pipeline of model operations—including training (pre-training, fine-tuning, alignment), inference, evaluation, and deployment—for over 350 LLMs and more than 90 multi-modal LLMs (MLLMs).\n",
    "\n",
    "Moreover, ms-swift is highly user-friendly. During training, each time the validation loss (also called evaluation loss) is computed, the framework automatically saves the current model parameters (model_checkpoint). At the end of training, it retains the version with the lowest validation loss, which corresponds to the best_model_checkpoint shown in the figure below.\n",
    "\n",
    "<div style=\"text-align: center;\">\n",
    "<img src=\"https://img.alicdn.com/imgextra/i3/O1CN0150XsFO1xM4z7CUMNr_!!6000000006428-2-tps-2288-136.png\" style=\"width: 70%;display: block; margin-left: auto; margin-right: auto\"/>\n",
    "</div>\n",
    "\n",
    "In the following experiments, we will focus on adjusting three key parameters:\n",
    "\n",
    "* learning_rate\n",
    "* LoRA rank (lora_rank)\n",
    "* num_train_epochs (number of training epochs)\n",
    "\n",
    "We will also switch datasets to demonstrate how LoRA fine-tuning can be applied across different tasks. Other parameter changes—such as increasing the batch size (batch_size) to reduce training time—are made solely to speed up experimentation and improve presentation; you do not need to focus on them.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a91a945d",
   "metadata": {},
   "source": [
    "#### 3.3.1 First experiment (takes about 1 minute)\n",
    "\n",
    "For the initial experiment, it is recommended that you start by fine-tuning the model using the following parameter settings. We’ll use a dataset of 100 problem-solution pairs generated by DeepSeek-R1 for training. This provides a solid foundation so that you can observe improvements through parameter optimization in later stages.\n",
    "\n",
    "| Parameter | Parameter Value |\n",
    "| --- | --- |\n",
    "| learning rate (learning_rate) | 0.1 |\n",
    "| LoRA Rank (lora_rank) | 4 |\n",
    "| Training Epochs (num_train_epochs) | 1 |\n",
    "| Dataset (dataset) | current directory/resources/2_4/train_100.jsonl |\n",
    "\n",
    "<br>\n",
    "\n",
    "> **Note**: You can adjust all parameters freely, but due to display effects and memory constraints, there are the following limitations:\n",
    "  - batch_size <= 16 (memory constraint) \n",
    "  - max_length <= 512 (maximum length of each training data, memory constraint) \n",
    "  - lora_rank <= 64 (LoRA rank, memory constraint) \n",
    "  - eval_step <= 20 (for convenience of display) \n",
    "\n",
    "Start the experiment:\n",
    "\n",
    "The fine-tuning module of the ms-swift framework uses LoRA fine-tuning by default, so there is no need to explicitly specify the fine-tuning method in this experiment.\n",
    "\n",
    "At the same time, the framework automatically applies a learning rate decay strategy during training, gradually reducing the effective learning rate. This helps prevent the model from overshooting the optimal solution and improves convergence stability.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5988325c",
   "metadata": {
    "ExecutionIndicator": {
     "show": true
    },
    "execution": {
     "iopub.execute_input": "2025-07-15T05:39:43.785953Z",
     "iopub.status.busy": "2025-07-15T05:39:43.785613Z",
     "iopub.status.idle": "2025-07-15T05:40:08.196115Z",
     "shell.execute_reply": "2025-07-15T05:40:08.195552Z",
     "shell.execute_reply.started": "2025-07-15T05:39:43.785931Z"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "%env CUDA_VISIBLE_DEVICES=0\n",
    "%env LOG_LEVEL=INFO\n",
    "!swift sft \\\n",
    "--learning_rate '0.1' \\\n",
    "--lora_rank 4 \\\n",
    "--num_train_epochs 1 \\\n",
    "--dataset './resources/2_7/train_100.jsonl' \\\n",
    "--batch_size '8' \\\n",
    "--max_length 512 \\\n",
    "--eval_step 1 \\\n",
    "--model_type 'qwen2_5-1_5b-instruct' \\\n",
    "--model_id_or_path './model'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a52b2e2e",
   "metadata": {},
   "source": [
    "| Training loss image | Evaluation loss image |\n",
    "| --- | --- |\n",
    "|<img src=\"https://img.alicdn.com/imgextra/i2/O1CN0122CqML1xiykiTglmo_!!6000000006478-2-tps-667-451.png\" style=\"width: 500px;display: block; margin-left: auto; margin-right: auto\"/> | <img src=\"https://img.alicdn.com/imgextra/i4/O1CN01AxXE0V1JqEORoVBdi_!!6000000001079-2-tps-667-451.png\" style=\"width: 500px;display: block; margin-left: auto; margin-right: auto\"/> |\n",
    "\n",
    "| **Observation metrics (training loss, validation loss):** | Training loss increases, validation loss increases |\n",
    "| --- | --- |\n",
    "| **Training status:** | **Training failed** |\n",
    "| **Cause analysis:** | The learning rate is likely too high. This causes the model's parameters to oscillate wildly instead of converging toward the optimal solution, resulting in training failure.<img src=\"https://img.alicdn.com/imgextra/i1/O1CN01l4leTB1LKI0BcVs16_!!6000000001280-2-tps-1658-1262.png\" style=\"width: 300px;display: block; margin-left: auto; margin-right: auto\"/>|\n",
    "| **Adjustment logic:** | Significantly reduce the learning rate to $0.00005$, allowing the model to \"learn cautiously\" with smaller steps. |\n",
    "\n",
    "#### 3.3.2 Second experiment (takes about 2 minutes)\n",
    "\n",
    "<div style=\"width: 30%\">\n",
    "    \n",
    "| Parameter | Old parameter value | New parameter value |\n",
    "| --- | --- | --- |\n",
    "| Learning rate (learning_rate) | 0.1 | 0.00005 |\n",
    "    \n",
    "</div>  \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "053c0d3d",
   "metadata": {
    "ExecutionIndicator": {
     "show": true
    },
    "execution": {
     "iopub.execute_input": "2025-07-15T05:40:24.914984Z",
     "iopub.status.busy": "2025-07-15T05:40:24.914644Z",
     "iopub.status.idle": "2025-07-15T05:40:49.090159Z",
     "shell.execute_reply": "2025-07-15T05:40:49.089605Z",
     "shell.execute_reply.started": "2025-07-15T05:40:24.914963Z"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "%env CUDA_VISIBLE_DEVICES=0\n",
    "%env LOG_LEVEL=INFO\n",
    "!swift sft \\\n",
    "--learning_rate '0.00005' \\\n",
    "--lora_rank 4 \\\n",
    "--num_train_epochs 1 \\\n",
    "--dataset './resources/2_7/train_100.jsonl' \\\n",
    "--batch_size '8' \\\n",
    "--max_length 512 \\\n",
    "--eval_step 1 \\\n",
    "--model_type 'qwen2_5-1_5b-instruct' \\\n",
    "--model_id_or_path './model'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6af933c4",
   "metadata": {},
   "source": [
    "| Training loss image | Evaluation loss image |\n",
    "| --- | --- |\n",
    "|<img src=\"https://img.alicdn.com/imgextra/i3/O1CN01DgtNVX1EDgzHYamOE_!!6000000000318-2-tps-680-451.png\" style=\"width: 500px;display: block; margin-left: auto; margin-right: auto\"/> | <img src=\"https://img.alicdn.com/imgextra/i3/O1CN01621v4k1ErzqC24Z1b_!!6000000000406-2-tps-689-451.png\" style=\"width: 500px;display: block; margin-left: auto; margin-right: auto\"/> |\n",
    "\n",
    "| **Observation metrics (Training loss, Validation loss):** | Training loss decreases, validation loss also decreases |\n",
    "| --- | --- |\n",
    "| **Training status:** | **Underfitting** |\n",
    "| **Cause analysis:** | Underfitting is a very common phenomenon during training. It indicates that, with the parameters unchanged, simply allowing the model to train longer can lead to successful training. Of course, modifying the parameters can also accelerate the training process. |\n",
    "| **Adjustment logic:** | 1. Allow the model to train longer by increasing the number of epochs `num_train_epochs` to 50. <br/> 2. Increase the `batch_size` to 16 to accelerate the training process. |\n",
    "\n",
    "#### 3.3.3 Third experiment (takes about 10 minutes)\n",
    "\n",
    "<div style=\"width: 50%\">\n",
    "\n",
    "| Parameter | Old Parameter Value | New Parameter Value |\n",
    "| :--- | :--- | :--- |\n",
    "| Number of Training Epochs (num_train_epochs) | 1 | 50 |\n",
    "| batch_size | 8 | 16 |\n",
    "| eval_step | 1 | 20 (Optimized output display) |\n",
    "\n",
    "</div> \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "647638fe",
   "metadata": {
    "ExecutionIndicator": {
     "show": true
    },
    "execution": {
     "iopub.execute_input": "2025-07-15T05:43:16.687835Z",
     "iopub.status.busy": "2025-07-15T05:43:16.687479Z",
     "iopub.status.idle": "2025-07-15T05:44:31.312976Z",
     "shell.execute_reply": "2025-07-15T05:44:31.312405Z",
     "shell.execute_reply.started": "2025-07-15T05:43:16.687813Z"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "%env CUDA_VISIBLE_DEVICES=0\n",
    "%env LOG_LEVEL=INFO\n",
    "!swift sft \\\n",
    "--learning_rate '0.00005' \\\n",
    "--lora_rank 4 \\\n",
    "--num_train_epochs 50 \\\n",
    "--dataset './resources/2_7/train_100.jsonl' \\\n",
    "--batch_size '16' \\\n",
    "--max_length 512 \\\n",
    "--eval_step 20 \\\n",
    "--model_type 'qwen2_5-1_5b-instruct' \\\n",
    "--model_id_or_path './model'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8933c14b",
   "metadata": {},
   "source": [
    "| Training loss image | Evaluation loss image |\n",
    "| --- | --- |\n",
    "|<img src=\"https://img.alicdn.com/imgextra/i4/O1CN01xsw3a31YarKvsEKCR_!!6000000003076-2-tps-671-451.png\" style=\"width: 500px;display: block; margin-left: auto; margin-right: auto\"/> | <img src=\"https://img.alicdn.com/imgextra/i3/O1CN01b2v3fK1jOSNo73Q3y_!!6000000004538-2-tps-680-451.png\" style=\"width: 500px;display: block; margin-left: auto; margin-right: auto\"/> |\n",
    "\n",
    "| **Observation metrics (Training Loss, Validation Loss):** | Training loss decreases, validation loss first decreases then increases |\n",
    "| --- | --- |\n",
    "| **Training status:** | **overfitting** |\n",
    "| **Cause analysis:** | overfitting is also a very common phenomenon during training. It indicates that the model is \"memorizing questions\" and not learning the knowledge in the dataset. We can reduce the number of epochs or increase the amount of data to make the model \"forget the questions.\" |\n",
    "| **Adjustment logic:** | 1. Reduce the num_train_epochs to 3. <br/> 2. Expand the number of problem solutions generated by DeepSeek-R1 to 1000 entries. Dataset location: current directory/resources/2_4/train_1k.jsonl <br/> 3. After increasing the amount of data, increase the rank of LoRA to 8 based on previous learning. |\n",
    "\n",
    "In general, with the scale of today's LLMs, fine-tuning requires at least **1,000+** high-quality training dataset entries. When below this threshold, the model tends to \"memorize questions\" after a few rounds of training instead of learning the inherent knowledge within the data.\n",
    "\n",
    "#### 3.3.4 Fourth experiment (takes about 5 minutes)\n",
    "\n",
    "| Parameter | Old Value | New Value |\n",
    "| --- | --- | --- |\n",
    "| Change Dataset | 100 entries | 1000+ entries |\n",
    "| Number of Training Epochs (num_train_epochs) | 50 | 3 |\n",
    "| LoRA Rank (lora_rank) | 4 | 8 (For reasons why this was increased, refer to the LoRA introduction). | \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a77d9f59",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-07-15T08:24:49.164123Z",
     "iopub.status.busy": "2025-07-15T08:24:49.163788Z",
     "iopub.status.idle": "2025-07-15T08:30:20.768915Z",
     "shell.execute_reply": "2025-07-15T08:30:20.768351Z",
     "shell.execute_reply.started": "2025-07-15T08:24:49.164103Z"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "%env CUDA_VISIBLE_DEVICES=0\n",
    "%env LOG_LEVEL=INFO\n",
    "!swift sft \\\n",
    "--learning_rate '0.00005' \\\n",
    "--lora_rank 8 \\\n",
    "--num_train_epochs 3 \\\n",
    "--dataset './resources/2_7/train_1k.jsonl' \\\n",
    "--batch_size '16' \\\n",
    "--max_length 512 \\\n",
    "--eval_step 20 \\\n",
    "--model_type 'qwen2_5-1_5b-instruct' \\\n",
    "--model_id_or_path './model'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f62bd2ca",
   "metadata": {},
   "source": [
    "| Training loss image | Evaluation loss image |\n",
    "| --- | --- |\n",
    "|<img src=\"https://img.alicdn.com/imgextra/i3/O1CN01p8rX0d1UAyUOGHeOJ_!!6000000002478-2-tps-671-451.png\" style=\"width: 500px;display: block; margin-left: auto; margin-right: auto\"/> | <img src=\"https://img.alicdn.com/imgextra/i1/O1CN01LjmbJ21P4Uo8ZJyav_!!6000000001787-2-tps-689-451.png\" style=\"width: 500px;display: block; margin-left: auto; margin-right: auto\"/> |\n",
    "\n",
    "\n",
    "| **Observation Metrics (Training Loss, Validation Loss):** | Training loss decreases, validation loss also decreases |\n",
    "| --- | --- |\n",
    "| **Training Status:** | **Underfitting** |\n",
    "| **Cause Analysis:** | The model is still underfitting, but the steady decrease in both training and validation loss indicates that the training process is stable and effective. We are on the right track. |\n",
    "| **Adjustment Logic:** | Allow the model to continue learning by increasing the number of training epochs (num_train_epochs) to 15. |\n",
    "\n",
    "#### 3.3.5 Fifth experiment (takes about 20 minutes)\n",
    "\n",
    "| Parameter | Old Parameter Value | New Parameter Value |\n",
    "| --- | --- | --- |\n",
    "| num_train_epochs | 3 | 15 |\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bbae8bc9",
   "metadata": {
    "ExecutionIndicator": {
     "show": true
    },
    "execution": {
     "iopub.execute_input": "2025-07-15T07:08:25.039405Z",
     "iopub.status.busy": "2025-07-15T07:08:25.039065Z",
     "iopub.status.idle": "2025-07-15T07:34:57.923017Z",
     "shell.execute_reply": "2025-07-15T07:34:57.922451Z",
     "shell.execute_reply.started": "2025-07-15T07:08:25.039384Z"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "%env CUDA_VISIBLE_DEVICES=0\n",
    "%env LOG_LEVEL=INFO\n",
    "!swift sft \\\n",
    "--learning_rate '0.00005' \\\n",
    "--lora_rank 8 \\\n",
    "--num_train_epochs 15 \\\n",
    "--dataset './resources/2_7/train_1k.jsonl' \\\n",
    "--batch_size '16' \\\n",
    "--max_length 512 \\\n",
    "--eval_step 20 \\\n",
    "--model_type 'qwen2_5-1_5b-instruct' \\\n",
    "--model_id_or_path './model'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2278eac9",
   "metadata": {},
   "source": [
    "| Training loss image | Evaluation loss image |\n",
    "| --- | --- |\n",
    "|<img src=\"https://img.alicdn.com/imgextra/i4/O1CN01hyQhbn1p04zyTeQkv_!!6000000005297-2-tps-671-451.png\" style=\"width: 500px;display: block; margin-left: auto; margin-right: auto\"/> | <img src=\"https://img.alicdn.com/imgextra/i3/O1CN01oy2oZv1r0ejEmpYdQ_!!6000000005569-2-tps-680-451.png\" style=\"width: 500px;display: block; margin-left: auto; margin-right: auto\"/> |\n",
    "\n",
    "\n",
    "| **Observation Metrics (Training Loss, Evaluation Loss):** | Training loss basically does not decrease, evaluation loss also basically does not decrease and even slightly increases |\n",
    "| --- | --- |\n",
    "| **Training Status:** | **Training Successful!** |  \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecf3c61e",
   "metadata": {},
   "source": [
    "### 3.4 Examination after fine-tuning\n",
    "\n",
    "After fine-tuning, two checkpoint files are typically saved:\n",
    "\n",
    "* best_model_checkpoint: The model parameters corresponding to the best performance on the validation set.\n",
    "* last_model_checkpoint: The model parameters at the end of the training process.\n",
    "\n",
    "In the code below, replace ckpt_dir with the path to the best_model_checkpoint to load the optimal version of the fine-tuned model.\n",
    "\n",
    "First, let's load the model into memory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "6db20cca",
   "metadata": {
    "ExecutionIndicator": {
     "show": false
    },
    "execution": {
     "iopub.execute_input": "2025-07-15T08:33:00.584525Z",
     "iopub.status.busy": "2025-07-15T08:33:00.584196Z",
     "iopub.status.idle": "2025-07-15T08:33:00.833205Z",
     "shell.execute_reply": "2025-07-15T08:33:00.832744Z",
     "shell.execute_reply.started": "2025-07-15T08:33:00.584505Z"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from swift.tuners import Swift\n",
    "\n",
    "# Please modify ckpt_dir to the correct location before running\n",
    "ckpt_dir = 'output/qwen2_5-1_5b-instruct/v9-20250715-150832/checkpoint-1035' # Modify to your checkpoint location before running\n",
    "# Load the model\n",
    "ft_model = Swift.from_pretrained(model, ckpt_dir, inference_mode=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94883c28",
   "metadata": {},
   "source": [
    "Let's take a look at how the fine-tuned model performs in the exam."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bff22696",
   "metadata": {
    "ExecutionIndicator": {
     "show": true
    },
    "execution": {
     "iopub.execute_input": "2025-07-15T08:33:04.539591Z",
     "iopub.status.busy": "2025-07-15T08:33:04.539275Z",
     "iopub.status.idle": "2025-07-15T08:35:30.293744Z",
     "shell.execute_reply": "2025-07-15T08:35:30.293287Z",
     "shell.execute_reply.started": "2025-07-15T08:33:04.539556Z"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "import json\n",
    "sum, score = 0, 0.0\n",
    "for line in open(\"./resources/2_7/test.jsonl\"):\n",
    "    # Read math questions from the test set\n",
    "    math_question = json.loads(line)\n",
    "    query = math_question[\"messages\"][1][\"content\"]\n",
    "    # Use the fine-tuned model for inference\n",
    "    response, _ = inference(ft_model, template, query)\n",
    "    # Get the correct answer\n",
    "    ans = math_question[\"messages\"][2][\"content\"]\n",
    "    pos = ans.find(\"ans\")\n",
    "    end_pos = ans[pos:].find('}}')\n",
    "    ans = ans[pos - 2: end_pos + pos + 2]\n",
    "    # Organize output\n",
    "    print((\"========================================================================================\"))\n",
    "    print(query.split(\"#Math Problem#\\n\")[1])\n",
    "    print(\"The answer to the question is: \" + ans)\n",
    "    print(\"-----------Model Response----------------\")\n",
    "    display(Latex(response))\n",
    "    print(\"-----------End of Response----------------\")\n",
    "    # Calculate the model's score\n",
    "    if ans in response:\n",
    "        score += 1\n",
    "        print(\"The model answered correctly\")\n",
    "    elif ans[6 : -2] in response:\n",
    "        score += 0.5\n",
    "        print(\"The model answered correctly but the output format was incorrect\")\n",
    "    else: print(\"The model answered incorrectly\")\n",
    "    sum += 1\n",
    "# Summary\n",
    "display(Markdown(\"The fine-tuned model scored **\" + str(int(100*score/sum)) + \"** points on the exam\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "960edb37",
   "metadata": {},
   "source": [
    "### 3.5 Parameter matrix fusion\n",
    "\n",
    "After the model training is completed, there are two ways to use the fine-tuned model:\n",
    "\n",
    "1. Dynamically load the fine-tuned LoRA weights at inference time. <br>\n",
    "      The low-rank parameter matrix obtained from fine-tuning takes up only about 20 MB of storage space, making it highly efficient for incremental deployment and distribution. This is a commonly used approach in engineering practice. <br>\n",
    "      Note: Whichever base model was used for fine-tuning, you must specify the correct base model when loading. <br>\n",
    "      In the previous subsection, we already used this method by setting the `ckpt_dir`.\n",
    "2. Merge the base model with the fine-tuned low-rank parameters into a single complete model, then deploy the merged model.\n",
    "\n",
    "Here, we introduce the second method: combining the \"fine-tuned parameter matrix\" (such as LoRA weights) with the \"base model parameter matrix\" to create a standalone model with updated parameters.\n",
    "\n",
    "By using the `swift export` command and providing the path to the fine-tuned model (preferably the `best_model_checkpoint`), you can generate the merged model. This exported model can be used independently without requiring special PEFT libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01928dad",
   "metadata": {
    "ExecutionIndicator": {
     "show": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "%env LOG_LEVEL=INFO\n",
    "!swift export \\\n",
    "--ckpt_dir 'output/qwen2_5-1_5b-instruct/vx-xxx/checkpoint-xxx<Modify to checkpoint location before running>' \\\n",
    "--merge_lora true"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb781cd7",
   "metadata": {},
   "source": [
    "The log displays the path of the model after fusion. By default, the complete parameter matrix after merging is saved in the `checkpoint` directory.\n",
    "\n",
    "(For the PAI experimental environment, the full model parameters are located at: `output/qwen2_5-1_5b-instruct/vX-XXX/checkpoint-XX-merged`)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70a31b63",
   "metadata": {},
   "source": [
    "## ✅ Summary\n",
    "\n",
    "In this lesson, we have learned the following:\n",
    "\n",
    "* Understanding the core value of model fine-tuning: By injecting task-specific data (such as math problem solutions), fine-tuning directly enhances the model’s reasoning ability in targeted domains, overcoming the limitations of prompt engineering and RAG-based chatbots.\n",
    "* Mastering key training parameters:\n",
    "    * Learning rate controls the step size of parameter updates.\n",
    "    * Epoch determines how many times the model iterates over the entire dataset.\n",
    "    * Batch size affects gradient stability and memory usage.\n",
    "* The loss function provides feedback on training progress by measuring prediction error.\n",
    "* Understanding the principle of LoRA efficient fine-tuning: LoRA reduces memory and computational costs through low-rank matrix decomposition—updating only small auxiliary matrices instead of all model parameters. In practice, adjusting the lora_rank parameter allows trade-offs between model capacity and training efficiency.\n",
    "* Completing iterative hyperparameter tuning experiments: Through multiple rounds of adjustment—such as modifying learning rate, data volume, and number of training epochs—we addressed underfitting and overfitting, ultimately achieving a significant improvement in the model’s problem-solving accuracy.\n",
    "\n",
    "Although this tutorial allows you to use prepared datasets and experience fine-tuning with free GPU resources, in real-world production scenarios, fine-tuning is not straightforward. It requires careful consideration of factors such as computational costs, data scale, and data quality. In particular, pay attention to the following:\n",
    "\n",
    "1. Assess whether simpler, lower-cost methods—like prompt engineering or RAG chatbots—are sufficient for the task before proceeding to fine-tuning.\n",
    "2. Ensure that the amount and quality of labeled data meet the minimum threshold—typically at least 1,000 high-quality, task-relevant samples—to achieve meaningful improvements.\n",
    "3. Verify that the project budget aligns with required expertise and infrastructure, ensuring acceptable cost-effectiveness.\n",
    "\n",
    "### Further learning\n",
    "#### Fine-tuning for more machine learning tasks\n",
    "\n",
    "* Image Classification\n",
    "    * *Examples*: object recognition, medical image diagnosis\n",
    "    * *Purpose*: Adapt pre-trained models (examples: ResNet, ViT) to extract features specific to a new image dataset.\n",
    "    * *Key points*: Leverage general visual knowledge from pre-training; reduce data requirements through transfer learning.\n",
    "* Object Detection\n",
    "    * *Examples*: autonomous driving, security monitoring\n",
    "    * Purpose: Fine-tune models like YOLO or Faster R-CNN to detect specific objects or scenes.\n",
    "    * *Key points*: Improve sensitivity to target categories and locations, reducing false alarms and missed detections.\n",
    "* Machine Translation \n",
    "    * *Examples*: domain-specific translation, customer support localization\n",
    "    * *Purpose*: Adapt general-purpose translation models (such as mBART or T5) to professional terminology and stylistic conventions.\n",
    "    * *Key points*: Correct semantic biases that arise when general models are applied to specialized domains.\n",
    "* Recommendation Systems\n",
    "    * *Examples*: e-commerce, content platforms\n",
    "    * *Purpose*: Optimize recommendation models (such as collaborative filtering or deep ranking models) using user behavior data.\n",
    "    * *Key points*: Balance personalization with cold-start challenges to improve click-through and conversion rates.\n",
    "\n",
    "#### More efficient fine-tuning methods\n",
    "\n",
    "* **Freeze**: One of the earliest PEFT methods. Most of the model’s parameters are frozen during training, and only a small portion—such as the final layers—are updated.\n",
    "    Characteristics:\n",
    "        * High parameter efficiency (only a small number of parameters are trained).\n",
    "        * Effective when the downstream task is similar to the pre-training objective (such as text classification).\n",
    "        * May underperform on complex or highly divergent tasks due to limited adaptability.\n",
    "<div style=\"text-align: left;\">\n",
    "<img src=\"https://img.alicdn.com/imgextra/i1/O1CN01X9GOk81sgAEtxflGR_!!6000000005795-2-tps-1340-686.png\" style=\"width: 600px;display: block; margin-left: 60px; margin-right: auto\"/>\n",
    "</div>\n",
    "\n",
    "* **Adapter Tuning**: Small neural network modules called adapter layers are inserted at specific positions within the original model architecture (such as between transformer layers). During fine-tuning, the original model parameters are frozen—only the Adapter layers are trained. \n",
    "    Characteristics:\n",
    "        * Modular design with strong compatibility across tasks.\n",
    "        * Slightly higher number of trainable parameters compared to LoRA, but delivers stable performance.\n",
    "        * Requires structural modifications to the model, and introduces additional computational overhead during inference due to the extra layers.\n",
    "<div style=\"text-align: left;\">\n",
    "<img src=\"https://img.alicdn.com/imgextra/i2/O1CN016gccCd1CdDpjDxbe9_!!6000000000103-2-tps-1482-1048.png\" style=\"width: 500px;display: block; margin-left: 60px; margin-right: auto\"/>\n",
    "</div>\n",
    "\n",
    "* Prompt Tuning: Indirectly controls model behavior by introducing learnable input vectors (called prompts) during training. The original model parameters are frozen, and only these prompt embeddings are updated. \n",
    "    Characteristics:\n",
    "        * No need to modify the model architecture—only the input representation is adjusted.\n",
    "        * Well-suited for generative tasks such as translation, dialog generation, and summarization.\n",
    "        * Performance depends heavily on prompt design; may underperform on complex or highly structured tasks.\n",
    "\n",
    "#### Fine-tuning dataset construction strategy\n",
    "\n",
    "In general, for more complex scenarios, effective fine-tuning requires at least **1,000+ high-quality training dataset samples**. When building your dataset, confirm the following key points:\n",
    "\n",
    "* **Data Quality**: Ensure the dataset is accurate, relevant, and free of ambiguity or errors. Remove noisy or incorrect entries that could mislead training.\n",
    "* **Diversity Coverage**: Include a wide range of scenarios, contexts, and domain-specific terminology to prevent the model from overfitting to a narrow data distribution.\n",
    "* **Class Balance**: For classification tasks with multiple categories, ensure balanced representation across classes to avoid bias toward dominant ones.\n",
    "* **Continuous Iteration**: Fine-tuning is an iterative process. Continuously optimize and expand the dataset based on the model's performance on the validation set.\n",
    "\n",
    "If you lack sufficient labeled data, consider enhancing the model’s knowledge using retrieval-augmented approaches, such as querying a knowledge base (business documents, FAQs).\n",
    "\n",
    "> In many real-world business scenarios, combining model fine-tuning with knowledge base retrieval yields better results than either method alone.\n",
    "\n",
    "You can also use the following strategies to expand the dataset:\n",
    "\n",
    "* **Manual Annotation**: Domain experts label or enrich data for critical or representative scenarios.\n",
    "* **Model Generation**: Use LLMs to generate synthetic data that mimics real-world usage.\n",
    "* **External Collection**: Gather data from public datasets, web scraping, user interactions, or customer feedback.\n",
    "\n",
    "#### Common evaluation metrics for models\n",
    "\n",
    "Evaluation metrics differ significantly depending on the task type. Below are some widely used metrics for common AI tasks:\n",
    "\n",
    "* **Classification tasks**:\n",
    "    * Accuracy: Proportion of correct predictions out of total predictions.\n",
    "    * Precision, Recall, and F1 Score: Especially useful in binary or multi-class classification to evaluate how well the model identifies positive instances.\n",
    "\n",
    "* **Text generation tasks**:\n",
    "    * BLEU (Bilingual Evaluation Understudy): Commonly used in natural language processing tasks such as machine translation; computes scores based on n-gram overlaps between generated text and reference translation.\n",
    "    * ROUGE (Recall-Oriented Understudy for Gisting Evaluation): Often used for summarization; measures recall of n-grams, precision, and F-measure.\n",
    "    * Perplexity: Evaluates how well a probability model predicts a sample. Lower perplexity indicates better performance.\n",
    "\n",
    "* **Image recognition/Object detection**:\n",
    "    * Intersection over union (IoU): Measures the overlap between predicted and ground-truth bounding boxes. A higher IoU indicates better localization accuracy.\n",
    "    * mAP (mean average precision): A standard metric in object detection that averages precision across different classes and confidence thresholds."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e76c79e-cb06-4a10-b6f0-57cc84ed069d",
   "metadata": {},
   "source": [
    "## 🔥 Quiz\n",
    "### 🔍 Multiple-Choice Question\n",
    "\n",
    "<details>\n",
    "<summary style=\"cursor: pointer; padding: 12px; border: 1px solid #dee2e6; border-radius: 6px;\">\n",
    "<b>Which of the following statements about LoRA is incorrect❓(Select 1.)</b>\n",
    "\n",
    "- A. LoRA can effectively reduce the cost of fine-tuning LLMs.\n",
    "- B. LoRA modifies the original weights of the fine-tuned model.\n",
    "- C. LoRA's implementation is relatively simple and easy to integrate.\n",
    "- D. The results of LoRA fine-tuning can be easily reverted.\n",
    "\n",
    "**[Click to View Answer]**\n",
    "</summary>\n",
    "\n",
    "<div style=\"margin-top: 10px; padding: 15px; border: 1px solid #dee2e6; border-radius: 0 0 6px 6px;\">\n",
    "\n",
    "✅ **Reference Answer: B**  \n",
    "📝 **Explanation**:  \n",
    "- LoRA does not directly modify the original weights but indirectly affects model behavior by adding low-rank matrices. This makes rollback operations simple, as you only need to remove the added low-rank matrices.\n",
    "\n",
    "</div>\n",
    "</details>\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "\n",
    "<details>\n",
    "<summary style=\"cursor: pointer; padding: 12px; border: 1px solid #dee2e6; border-radius: 6px;\">\n",
    "<b>You are using Swift to fine-tune a Qwen model and notice a significant upward trend in loss on the validation set. Which of the following actions can help alleviate or resolve this issue❓(Select all that apply.)</b>\n",
    "\n",
    "- A. Increase learning rate\n",
    "- B. Decrease learning rate\n",
    "- C. Increase --num_train_epochs\n",
    "- D. Decrease --num_train_epochs\n",
    "\n",
    "**[Click to View Answer]**\n",
    "</summary>\n",
    "\n",
    "<div style=\"margin-top: 10px; padding: 15px;  border: 1px solid #dee2e6; border-radius: 0 0 6px 6px;\">\n",
    "\n",
    "✅ **Reference Answer: BD**  \n",
    "📝 **Explanation**:  \n",
    "- learning_rate: A high learning rate can lead to fast model training but may cause oscillations near the optimal solution, or even non-convergence, resulting in fluctuating loss, which may appear like overfitting. However, this is different from true overfitting.  \n",
    "- num_train_epochs: Overfitting may also be caused by too many training epochs. Reducing the number of training epochs can prevent the model from over-learning the training data.\n",
    "\n",
    "</div>\n",
    "</details>  \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edcef42f",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llm_learn",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
